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Mr.  Chairman  and  Members  of  the  Subcommittee: 


We  are  pleased  to  discuss  the  results  of  our  work  for  today's 
hearing  on  the  National  Science  Foundation  (NSF )  April  1987  study 
entitled  "The  Science  and  Engineering  Pipeline."  As  you 
requested,  our  work  focused  on  the  type  and  level  of  review  the 
study  received  before  it  was  issued.  More  specifically,  we  (1) 
examined  how  the  study  was  reviewed  within  NSF  and  (2)  analyzed 
information  that  NSF  provided  in  a  September  13,  1991,  letter  to 
the  Subcommittee  on  the  external  review  process  used  for  the  study. 
As  agreed  with  your  office,  this  completes  our  work  on  your 
request. 

In  summary,  we  found  that  the  April  1987  study  was  approved  by 
several  levels  of  officials  within  NSF  and  the  internal  review  it 
received  appears  to  have  followed  existing  NSF  procedures. 

However,  information  that  NSF  reported  with  its  September  1991 
letter  may  have  conveyed  the  impression  that  the  study  received 
more  formal  external  review  than  we  were  able  to  verify.  For 
example,  the  letter  states  that  the  study  was  reviewed  by  nine 
specific  individuals,  seven  from  outside  of  NSF  and  two  from 
another  NSF  division.  However,  we  contacted  these  nine  individuals 
and  found  that  eight  of  them  had  not  provided  formal  review  in  the 
form  of  written  or  oral  comments  to  NSF .  The  ninth  could  not 
remember  if  he  had  or  had  not  provided  this  type  of  review.  When 
we  brought  this  information  to  NSF  officials'  attention,  they  told 
us  that  they  used  the  word  "review"  to  mean  "professional 
interaction"  which  includes  discussions  within  the  professional 
community  on  concepts  and  topics  related  to  the  study. 

Before  I  discuss  these  issues  in  more  detail,  let  me  provide 
some  background  information  on  the  study. 

BACKGROUND  AND  DEBATE  OVER  THE  "PIPELINE"  STUDY 

NSF's  Policy  Research  and  Analysis  Division  (PRA)  began 
analyzing  the  demand-supply  balance  for  scientists  and  engineers  in 
1984  at  the  request  of  the  White  House's  Office  of  Science  and 
Technology  Policy.  NSF  formally  published  the  results  of  this 
analysis  for  the  first  time  in  an  April  1987  study  entitled  "The 
Science  and  Engineering  Pipeline"  (PRA  Issue  Paper  87-2)  and 
distributed  it  to  over  2,000  individuals  on  PRA's  general  mailing 
list.  The  study  became  known  as  the  "pipeline  study"  because  it 
said  that  natural  scientists  and  engineers  flowed  through  a 
"pipeline"  from  undergraduate  education  to  professional  employment 
in  natural  science  and  engineering  occupations.  NSF  defined 
natural  scientists  as  excluding  behavioral  and  social  scientists. 

As  a  part  of  its  analysis,  PRA  projected  a  cumulative 
"shortfall"  of  692,000  bachelor  of  science  degrees  during  the 
period  from  1985  to  2010.  NSF  uses  the  term  shortfall  to  refer  to 
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the  difference  between  an  historical  or  expected  value  and  an 
actual  or  adjusted  expected  value.  Since  1987  the  results  of  PRA’s 
analysis  have  been  published  and  distributed  on  eight  other 
occasions  in  the  form  of  issue  papers,  working  drafts,  and  a  book. 

In  August  1991,  this  Subcommittee  sent  a  letter  to  NSF  that 
requested  information  in  response  to  questions  related  to  the  PRA 
projections.  NSF  responded  to  these  questions  in  a  letter  sent  in 
September  1991,  along  with  accompanying  documents. 

INTERNAL  REVIEW  OF  THE  PIPELINE  STUDY 

NSF  has  given  its  directorates  (NSF's  major  organizational 
units)  responsibility  for  the  technical  review  of  all  reports 
within  their  organization.  This  includes  assuring  that  the  reports 
offer  information  that  is  of  value  and  is  adequate,  timely  and 
concise.  However,  NSF  also  has  a  formal  organizational  publication 
clearance  process  to  assure  that  publications  are  consistent  with 
current  policies  and  do  not  involve  duplication  of  effort  or 
excessive  expenditures. 

We  found  that  the  pipeline  study  was  drafted  by  analysts 
within  the  Science  Innovation  Policy  Section  of  PRA  and  then 
approved  by  officials  in  charge  of  that  Section,  the  PRA  Division, 
and  the  Scientific,  Technological  and  International  Affairs 
Directorate.1  For  example,  the  Science  Innovation  Policy  Section 
Head,  PRA  Director,  and  Scientific,  Technological,  and 
International  Affairs  Assistant  Director  signed  the  clearance  sheet 
approving  publication  of  the  April  1987  study. 

Also,  according  to  the  PRA  Director,  others  within  the 
Directorate  reviewed  versions  of  the  pipeline  study.  NSF  documents 
show  that,  in  November  of  1985,  a  Senior  Staff  Associate  in  the 
Directorate  and  the  Section  Head  of  Scientific  and  Technical 
Personnel  Studies  in  the  Science  Resources  Studies  Division  of  the 
Directorate  provided  written  comments  to  PRA  regarding  the  study. 

The  NSF  Director  at  the  time  the  study  was  issued  told  us  that 
he  and  the  Deputy  Director  were  heavily  involved  in  the  development 
of  the  April  1987  pipeline  study.  For  example,  an  October  1,  1985, 
memo  refers  to  a  September  5,  1985  briefing  at  which  they 
discussed,  raised  questions,  and  made  suggestions  on  issues 
relating  to  the  pipeline  study.  The  Director  also  stated  that  the 
pipeline  study  was  discussed  in  briefings  before  the  National 
Science  Board  and  the  Board's  Education  and  Human  Resources 
Committee.  For  example,  meeting  minutes  indicate  that  on  November 


1The  National  Science  Foundation  has  since  been  reorganized.  All 
references  to  the  organization  of  NSF  contained  in  this  testimony 
are  based  on  the  diagram  in  appendix  I  which  represents  NSF's 
organization  at  the  time  of  the  study. 
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21,  1985,  PRA  staff  presented  a  report  on  PRA's  pipeline  study  to  a 
meeting  of  the  Education  and  Human  Resources  Committee  of  the 
National  Science  Board. 

The  pipeline  study  was  also  approved  for  publication  by  the 
Office  of  Legislative  and  Public  Affairs  Director,  the  Scientific, 
Technological,  and  International  Affairs  Assistant  Director,  and 
the  PRA  Director.  The  Director  of  the  NSF  Office  of  Legislative 
and  Public  Affairs  told  us,  however,  that  he  would  not  approve  the 
April  1987  version  of  the  pipeline  study  for  publication  as  an  NSF 
report  because  publication  of  a  policy  document  like  it  was 
unusual,  if  not  unprecedented  at  NSF.  He  said  that  documents  that 
NSF  directorates  publish  are  usually  publications  like  newsletters 
that  are  directed  at  grant  recipients.  Accordingly,  the  1987  study 
was  issued  as  a  PRA  report  with  a  disclaimer  stating  that  it  did 
not  represent  the  official  policy  of  NSF  or  PRA. 

EXTERNAL  REVIEW  OF  THE  STUDY 

In  its  September  1991  letter  to  the  subcommittee,  NSF  listed 
seven  individuals  employed  outside  the  Foundation  and  two  NSF 
Science  Resource  Studies  employees  as  reviewers  of  the  April  1987 
study.  We  found  no  documentation  indicating  that  these  individuals 
provided  formal  comments  to  NSF  on  the  study.  When  we  contacted 
these  nine  individuals,  eight  of  them  told  us  that  they  had  not 
provided  formal  review  in  the  form  of  written  or  oral  comments  to 
the  PRA  staff  on  the  April  1987  study.  One  of  the  eight  said  that 
he  had  discussed  the  model  used  in  the  study  with  PRA  staff,  three 
said  that  they  may  have  had  informal  discussions  with  PRA  staff 
related  to  the  study  prior  to  the  study's  publication  in  April  1987 
and  four  indicated  they  had  received  copies  of  the  paper  but  had 
not  provided  any  written  or  oral  comments  to  PRA.  The  ninth 
individual  could  not  remember  if  he  had  or  had  not  provided  this 
type  of  review.  When  we  brought  this  information  to  NSF's 
attention,  officials  told  us  that  they  used  the  word  "review"  to 
mean  "professional  interaction"  which  includes  discussions  within 
the  professional  community  on  concepts  and  topics  related  to  the 
study. 

In  its  1991  reply  to  the  subcommittee,  NSF  also  listed  the 
participants  of  a  1986  PRA  workshop  as  "reviewers."  The  workshop 
focused  on  the  presentation  and  discussion  of  the  papers  NSF  had 
commissioned  on  five  topics  related  to  the  production  of  natural 
scientists  and  engineers  over  the  next  25  years.  However,  an  NSF 
staff  analyst  responsible  for  organizing  the  workshop  told  us  that 
the  pipeline  study  itself  was  not  reviewed  or  discussed  by  the 
participants  of  the  workshop. 

As  further  evidence  of  external  review  in  its  reply  to  the 
Subcommittee,  NSF  said  "the  Science  and  Engineering  Pipeline"  study 
was  submitted  to  the  Office  of  Technology  Assessment  (OTA)  as  input 
to  its  June  1988  Educating  Scientists  and  Engineers:  Grade  School 
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to  Grad  School  report.  Specifically,  NSF  said  that  "[a]s  a  part  of 
the  OTA  report,  the  PRA  contributions  were  reviewed  by  several  OTA 
staff  and  consultants  and  approved  by  an  advisory  panel  of  eminent 
experts  on  scientific  and  engineering  personnel  from  industrial  and 
academic  sectors." 

However,  according  to  the  OTA  project  director  for  the  June 
1988  report,  OTA  did  not  review  or  provide  comments  to  NSF  on  the 
study.  Moreover,  he  said  an  OTA  advisory  panel  did  not  approve  the 
pipeline  study  because  that  is  not  a  function  of  OTA  advisory 
panels . 


In  summary,  internal  review  of  the  April  1987  study  appears  to 
have  been  consistent  with  NSF  requirements.  Officials  within  the 
NSF  Scientific,  Technological,  and  International  Affairs 
Directorate  approved  the  publication  of  the  April  1987  study  and 
aspects  of  the  study  were  discussed  in  briefings  before  top  level 
NSF  officials  and  the  National  Science  Board  prior  to  its 
publication . 

However,  the  study  appears  to  have  received  little  or  no 
formal  review  from  persons  outside  NSF.  Further,  NSF ' s  September 
1991  letter  may  have  conveyed  the  impression  that  the  study 
received  more  external  review  than  we  could  verify. 

This  concludes  my  statement.  I  would  be  happy  to  respond  to 
any  questions  you  or  members  of  the  Subcommittee  may  have. 
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APPENDIX  I  APPENDIX  I 

Organizational  Chart  of  NSF 
Directorates,  Divisions  and  Sections 
Involved  in  Preparing  and  Reviewing 
the  April  1987  Study 
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